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Distal camera with panorama or mosaic functionality 



FIELD OF THE INVENTION 

The invention relates to an electronics device with a digital camera, and to a 
method of enabling to create a composite picture using a digital camera. 

BACKGROUND ART 

5 The digital, solid-state cameras are rapidly replacing the traditional analog 

cameras. Prices have come down to a mass-market level and features, unheard of in the 
conventional analog domain, are being added to give the user ever more control over the 
capturing process and the quality of the end-result. Currently, digital cameras are 
commercially available that have been designed for connecting to a PC in order to transfer 

10 image files. These files can include still image files (e.g., JPEG files) as well as motion video 
files (e.g., QuickTime files). Such a digital camera captures digital still images and motion 
video segments and stores the images using, e.g.. Flash EPROM memory. Digital cameras 
also are increasingly more fi^quently integrated with other appliances. Devices such as 
mobile phones currently come equipped with camera chips as a standard feature. Market 

15 reports estimate that in 2002 about 20 million (20M) so-called "camera phones" were sold 
worldwide. The embedded digital camera enables to instantly share a snapshot with a remote 
party, A similar trend of designed-in camera chips has appeared in the PDA market. Solid- 
state cameras thus have become a commodity in the consumer electronics (CE) environment, 
i.e., the mass market, wherein brand owners try to distinguish their products by means of 

20 attractive features providing additional functionality. 

An example of the above is discussed in US patent application publication 
20030095193. This reference discloses creating a panoramic digital image by providing a 
digital camera having a memory and which is operable in a first mode for producing 
individual still digital images, and in a second mode for capturing a series of overlapping 

25 digital images to be used in constructing the panoramic digital image. The digital camera is . 
mounted on a stand, rotated on the stand.through a series of predetermined positions, and 
operated in the second mode to capture the series of overlapping digital images. The series of 
overl£q)ping digital images corresponding to the panoramic digital image is stored in a 
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predetermined location in the memory, and processed to stitch such images together to 
produce the panoramic digital image. 

SUMMARY OF THE INVENTION 
5 The known camera requires that the user accurately follow a specific 

procedure to create a panoramic image. In addition, the camera needs a tripod or other stable 
stand. The mventors propose a user-£riendlier altemative to the known digital camera for 
creating a panoramic image. In order to capture an image of an object, e.g., a scene or a 
building facade, the inventors propose that the user scan the object with a single sweep of tiie 

10 camera while camera is taking pictures. Upon completing the sweep, the camera stitches the 
pictures taken so as to create a panorama or mosaic picture. 

More specifically, the invention relates to a digital camera having a memory 
and an operational mode wherein the camera takes a sequence of still pictures. A next one of 
the pictures in the sequence is selected for being stored in the memory based on an amount of 

1 5 overlap regarding a picture content with a previous one of the pictures stored in the memory. 
The camera processes the pictures stored in the memory so as to create a composite picture^ 
e.g., a panorama or mosaic picture. Preferably, the camera comprises a sensor for 
determining information representative of respective relative coordinates of the camera when i. 
taking respective ones of the pictures during the sweep. Such a sensor comprises, e.g., an 

20 accelerometer or tachometer from whose output the camera's positions can be obtained. The 
information can then be used for controlling the creating of the composite picture, e.g., if the 
sweep traces out a serpentine path to scan an object that is high as well as wide. The 
information can also be used to determine the next picture, for example in conjunction with 
information about the estimated distance between the camera and the relevant area of the 

25 scanned object. ITie latter distance can be automatically estimated based on, e.g., the focal 
settings of the camera with auto-focus functionality. Preferably, the camera comprises a 
detector for detecting an area in the composite picture lacking coverage by the sequence of 
pictures, e.g., for providmg feedback to a user of the camera. If the camera has a viewer to let 
the user examine a picture after it has been taken, a reduced version of the composite picture 

30 created thus far can be rendered as well. Any gap is then easily spotted so that the user can 
take an additional picture to cover the gap. The user is guided to point the camera to the 
proper area, e.g,, by overlaying in the viewer the real-time image, i.e., the image resulting 
from capturing at what the camera is being pointed, vntti parts of one or more pictures 
already taken that neighbor the gap. 
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In a further embodiment of the invention, the camera as specified above is 
embedded, or otherwise functionally integrated, with an electronic apparatus such as a mobile 
phone or palmtop computer or PDA. 

One of the advantages of having the image processed at the camera is that the 
5 user can preview the result almost instantly in order to determine if the result is satisfactory. 
For example, with a two-dimensional sweep the user may have missed an area of the scatmed 
object, for which a solution is provided above. Another advantage is that the camera of the 
invention is self-sufficient. Downloading the pictures taken to a PC and post-processing them 
on the PC is not required anymore. In this manner a low-cost digital camera with panorama 

10 and mosaic functionality is provided. 

The invention also relates to a method of creating a composite picture using a 
digital camera. The method comprises taking a sequence of still pictures; and selecting a next 
one of the pictures in the sequence for being stored based on an amount of overlap of a 
picture content with a previous one of the pictures stored. Then the pictures stored are 

15 processed so as to create a composite picture. Preferably, the method comprises determining 
information representative of respective relative coordinates of the camera when taking ' 
respective ones of the pictures. The information is used for controlling flie creating of the 
composite picture using a stitching algorithm and/or to determine the next picture to be • 
stored. Preferably, the method comprises identi^ing an area in the composite picture lacking 

20 coverage by the pictures stored in the memory. 

For completeness, reference is made to WOO 195 109 (attorney docket 
TW000002) published for Rob Beuker and Marcel Breeuwer for METHOD AND 
APPARATUS FOR MERGING IMAGES INTO A COMPOSITE IMAGE. This patent 
document relates to an apparatus and a method for merging overlapping two-dimensional 

25 (2D) images, which are formed by an image pick-up device as projections of a three- 
dimensional (3D) scene. In particular, the merging includes image registration by projective 
transformation of one of the 2D images, the transformation being derived fix>m corresponding 
feature found in both images. In order to achieve improved accuracy and stability, the 
coordinates of the corresponding feature points are chosen or are translated so that, on 

30 average, the numerical ranges of coordinate values are minimized. The apparatus includes an 
appropriately configured image processor or computer with an attached image acquisition 
device, which in one embodiment, is a diagnostic x-ray. 
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BRIEF DESCRIPTION OF THE DRAWING 

The invention is explained in further detail, by way of example and with 
reference to the accompanying drawing wherein: 

Fig. 1 is a block diagram of a c£imera in the invention; and 
5 Figs. 2 and 3 form a flow diagram illustrating a method of the invention; 

Throughout the figures, same reference numerals indicate similar or corresponding features. 

DETAILED EMBODIMENTS 

Fig. 1 is a block diagram of a, preferably hand-held, digital camera 100 

10 according to the invention. The camera in this example is a still-picture digital camera as 
opposed to a video camera. Camera 100 comprises an image sensor 102, a cache memory 
104, a memoiy 106, a data processor 108 and a display monitor or viewer 110. Processor 108 
comprises a position/orientation sensor 1 12, an overlap controller 114, image stitchmg 
software 116 and a gap detector 118. 

15 Sensor 102 converts an optical image of an object into digital data and 

comprises a solid-state image-sensing element (not shovm), such as a charge-coupled device 
(CCD). In c£ise a CCD is used, sensor 102 includes an analog-to-digital converter (not 
shown) to convert the CCD's analog output signals to digital data. Sensor 102 is connected to 
cache 104 that stores the digital data from sensor 102. When the user depresses the camera's 

20 shutter button (not shown), processor 108 controls the CCD so as to perform a, so-called, all- 
pixel readout, wherein the data is temporarily stored in cache 104. From there, the data is 
transferred to memory 106 under control of processor 108. Memory 106 comprises, e.g., a 
removable memory card or a hard-disk drive (HDD) with a small form-factor, etc. Display 
110 lets the user view the image as captured by sensor 102, e.g., while focusing or finding the 

25 proper object, as well as an image stored in memory 106. Operation thus far is similar to 
known digital cameras. 

Camera 100 of the invention has an operational mode, referred to herein as 
"scan mode", wherein a sequence of images is captured that covers an area that is larger than 
can be captured in a single image using the camera's optical components (not shown here). 

30 To this end, the camera has to be moved and/or its orientation changed by the user so as to 
enable taking pictures that together cover the targeted area. Operation is as follows. 

In the scan mode, the user sweeps camera 100 along a path for which the 
accumulated field of view is to cover a certain area. For example, the user sweeps camera 
100 along a path from left to right tiiat in order to capture, e.g., a large articulated steam 



wo 2005/041564 



PCT/IB2004/052176 



locomotive on display at a museum, or from bottom to top to capture, e.g., a redwood tree. A 
first picture is taken that gets transferred to memory 106. During the sweep camera 100 keeps 
on taking pictures. The refresh rate of sensor 102 and cache 104 is such that multiple pictures 
can be taken during the sweep that overlap with the first picture given the typical speed at 
5 which a human can move his/her arm and wrist holding the camera. Overlap controller 114 
determines the overlap of these subsequent pictures with the first picture and controls the 
transfer to memoiy 106 of the picture that has a certain minimum, or otherwise desired, 
amount of overlap. This last picture becomes the second picture. Camera 100 keeps taking 
pictures and overlap recognizer 1 14 similarly selects with respect to the second picture the 

10 third picture to be transferred to memory 106, and so on to the last picture. The last picture 
relates to the moment at which tibie scan mode is terminated by means of, e.g., the user 
releasing a button (not shown) kept depressed during the sweep and/or stopping the 
movement of the camera housing. In a simple horizontal sweep the order of the pictures in 
memory 106 is linear. Conventional stitching algorithms 1 16, known in the field of image 

15 processing, are run on processor 108 and create a composite picture fi*om the series of 
overlapping pictures available from memory 106. 

Controller 114 controls the overlap between successive pictures. Controller 
1 14 is implemented, e.g., by an algorithm that seeks to match pixels of the successive 
pictures in the region of expected overlz^. If sensor 1 12 senses camera 100 being moved to . 

20 the right, and preferably also the current speed, and has also information about the scanned 
object's distance from the camera, the expected region of overlap can be minimized given the 
angle covered by the camera's lens (not shown). The distance previously mentioned can be 
estimated, based on, e.g., the focal settings. Sensor 112 comprises, e.g., a simple 
accelerometer, or a tachometer that determines the speed relative to the earth's magnetic field 

25 based on the Hall elBTect, etc. Alternatively, the relative velocity (virtual pixel displacement in 
a certain direction per imit of time) of pixels in a picture in cache 104 relative to the pixels in 
the last picture stored in memory 106 can be determined by means of subjecting subsequent 
pictures to a motion estimation algorithm known from video processing. As the amount of 
overlap depends on the velocity, the overlap can be determined. 

30 The amount of desired overlap is preferably progranunable, e.g., made 

automatically dependent on the lighting conditions, on the high-fi^uency components in the 
picture that represent details, or is user-selectable. 

As another example, the user lets camera 100 trace out a serpentine-like path 
that substantially lies in a vertical plane to cover, e.g., a &cade of an interesting structure or 
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an object from close by. The ordering of the pictures taken is a linear one, but the linear order 
is to be mapped onto a two-dimensional array forming the eventual composite picture. 
Stitching now occurs at all four sides of a picture not at the perimeter of the two-dimensional 
area. This means that in order for the stitching to work, the relative two-dimensional position 
5 of each of the pictures, successively selected for transferal to memory 106, needs to be 

determined. To this end, camera 108 is equipped with position or orientation sensor 1 12 that 
enables to determine the path traced by camera 100 or the focal point of camera 100 during 
the serpentine sweep. Sensor 1 12 comprises, e.g., simple accelerometers (for vertical and for 
horizontal accelerations of the camera body and preferably also for determining the relative 
10 changes in pan and tilt angles>and circuitry to twice integrate over time the acceleration 

valueis found to determine the path. The pan or tilt motion can be determmed using, e.g., pairs 
of accelerometers that register accelerations in opposite directions. Accordingly, a picture 
selected for transferal to memory 106 can get associated with a mark along this path 
including the relative vertical and horizontal coordinates. This enables to map the linear . 
1 5 ordering onto the two-dimensional array. 

Figs. 2 and 3 form a flow diagram illustrating a process 200 according to the 
invention. In a step 202, the user depresses a scan-mode button of camera 100, or otherwise 
causes camera 100 to enter the scan-mode, and starts the sweep. In a step 204, a first picture 
is taken and stored in memory 106. Camera 100 continues to take pictures. The next picture 
20 in cache 104 is accessed in a step 206 and compared in a step 208 with the first picture that is 
stored in memory 106. In a step 210, it is determined whether the picture currently in cache 
104 has an acceptable overlap with the first picture. For example, if the overlap is large, it 
may be better to take a next picture, i.e., a split second later, that gets loaded into cache 104 
to replace the previous picture buffered in cache 104. This helps to reduce the number of 
pictures needed for creating the composite image.and makes efficient use of the camera's 
onboard storage 106. If the overlap is unacceptable, process 200 returns to step 206. If the 
overlq) is acceptable, the picture currently in cache 104 is transferred to memory 106 in a 
step 212. In a step 214 it is determined if the sweep is completed. This is done, for example, 
by registering that the sweeping motion has stopped or that the camera's scan-mode button 
has been released. Alternatively, the user may have set the maximum number of pictures to 
be saved to memory 106 in a single sweep, etc. If the sweep is still ongoing, process 200 
returns to step 206. If the sweep has finished, process 200 proceeds to a step 216 wherein the 
pictures accumulated in memory 106 are stitched to create the composite image, as discussed 
above. In a step 218, it is determined whether the composite image has one or more gaps, i.e.. 
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whether there are extended areas in the composite image wherein pixel data is lacking. If the 
areas are not extended in the sense that, e.g., simple interpolation from the area's perimeter 
can fill in the missing pixels (can be made a sub-routine of step 216), or if there are no gaps, 
process 200 is terminated in a step 220. If there is a gap process 200 goes to a step 222, 
5 wherein the user is guided to the missing area using, e.g., the overlay procedure mentioned 
above. Then, an extra picture is taken covering the missing area and transferred to memoiy 
106. Thereupon, process 200 returns to step 216 to integrate the extra picture with the 
composite image created thus far. Note that in the overlay procedure the location of the extra 
picture, that is to be integrated, is already known. 
10 As an alternative to starting the stitching together of the pictures in memory 

upon terminating the sweep, processor 108 decides on the coordinates of the pictures stored 
in memoiy 106 whether some or all of the pictures can akeady be stitched together during the 
sweep. 



